test: add nightly run flow for system tests by pnarayanaswamy · Pull Request #29484 · MetaMask/metamask-mobile

pnarayanaswamy · 2026-04-29T12:23:44Z

Description

Adds a new Nightly System Tests GitHub Actions workflow (run-system-tests.yml) that runs on a daily cron or manually, optionally accepting pre-built BrowserStack app URLs.

The workflow builds two RC variants (clean + with pre-imported SRP), uploads resulting APK/IPA artifacts to BrowserStack, runs Android/iOS login and onboarding test suites via BrowserStack Local, publishes test report artifacts, and fails the run if any platform flow fails.

Changelog

CHANGELOG entry:

Related issues

Fixes:

Manual testing steps

Feature: my feature name

  Scenario: user [verb for user action]
    Given [describe expected initial app state]

    When user [verb for user action]
    Then [describe expected outcome]

Screenshots/Recordings

Before

After

Pre-merge author checklist

I've followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
I've completed the PR template to the best of my ability
I've included tests if applicable
I've documented my code using JSDoc format if applicable
I've applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Performance checks (if applicable)

I've tested on Android
- Ideally on a mid-range device; emulator is acceptable
I've tested with a power user scenario
- Use these power-user SRPs to import wallets with many accounts and tokens
I've instrumented key operations with Sentry traces for production performance metrics
- See trace() for usage and addToken for an example

For performance guidelines and tooling, see the Performance Guide.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
Adds a new scheduled GitHub Actions workflow that builds and uploads signed RC artifacts and runs device tests on BrowserStack, which may affect CI load and secret usage if misconfigured.

Overview
Adds a new Nightly System Tests GitHub Actions workflow (.github/workflows/run-system-tests.yml) that runs on a daily cron or manually, builds clean and with-SRP RC variants, uploads APK/IPA artifacts to BrowserStack, and executes Android/iOS login + onboarding system test suites via BrowserStack Local with report artifacts and a failing summary gate.

Extends builds.yml with a new main-rc-with-srp build variant (RC config plus injected ADDITIONAL_SRP_1 and PREDEFINED_PASSWORD) to support login-focused system tests, and updates .github/CODEOWNERS to assign QA ownership for the new workflow.

^{Reviewed by Cursor Bugbot for commit 809f391. Bugbot is set up for automated code reviews on this repo. Configure here.}

github-actions · 2026-04-29T12:23:59Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

jvbriones · 2026-04-29T15:09:47Z

+    steps:
+      - name: Download Android APK (with-SRP)
+        if: needs.build-with-srp.result == 'success'
+        uses: actions/download-artifact@v4


jvbriones · 2026-04-29T15:11:13Z

+      BROWSERSTACK_ANDROID_APP_URL: ${{ needs.upload-to-browserstack.outputs.android-login-url }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4


jvbriones · 2026-04-29T15:11:22Z

+        uses: actions/checkout@v4
+
+      - name: Restore node_modules cache
+        uses: actions/cache@v4


v4 is deprecated

tommasini · 2026-05-04T16:20:27Z

    "build:android:main:prod": "./scripts/build.sh android main production",
    "build:android:main:beta": "./scripts/build.sh android main beta",
    "build:android:main:rc": "./scripts/build.sh android main rc",
+    "build:android:main:rc:with:srp": "./scripts/build.sh android main rc",


Why we are not reusing the existing build:android:main:rc?

This doesn't seem to be used

this build is added because it includes the srp so we don't have to go through the onboarding flow during the tests

github-actions · 2026-05-07T14:49:29Z

🔍 Smart E2E Test Selection

Selected E2E tags: None (no tests recommended)
Selected Performance tags: None (no tests recommended)
Risk Level: low
AI Confidence: 97%

click to see 🤖 AI reasoning details

E2E Test Selection:
All three changed files are purely CI/infrastructure changes with no impact on application source code or existing test suites:

.github/workflows/run-system-tests.yml — A new GitHub Actions workflow file that adds a "Nightly System Tests" pipeline. It runs on a schedule (5 AM UTC daily) or manually via workflow_dispatch. It does NOT trigger on PRs, so it won't affect PR-level E2E test runs. The workflow orchestrates BrowserStack-based system tests for login and onboarding flows, but this is a separate test infrastructure from Detox E2E tests.
.github/CODEOWNERS — A single-line addition assigning @MetaMask/qa ownership to the new workflow file. No functional impact.
builds.yml — Adds a new main-rc-with-srp build configuration that bakes in a pre-imported wallet for system tests. This is an additive change that doesn't modify any existing build configurations, so it cannot break existing builds or tests.

None of these changes touch: app source code, controllers, UI components, navigation, test fixtures, page objects, or Detox test files. There is no risk to any existing E2E test flows. No Detox tags need to run for this PR.

Performance Test Selection:
No application code, performance-sensitive components, or performance test infrastructure was modified. The changes are purely CI workflow and build configuration additions. No performance tests are warranted.

View GitHub Actions results

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 809f391. Configure here.}

cursor · 2026-05-07T14:53:02Z

+        uses: actions/checkout@v6
+
+      - name: Restore node_modules cache
+        uses: actions/cache@v6


Non-existent action versions break all test jobs

High Severity

actions/cache@v6 and actions/upload-artifact@v8 do not exist as released versions. The latest available are actions/cache@v5 and actions/upload-artifact@v7 (while actions/download-artifact@v8 and actions/checkout@v6 do exist — they have different version tracks). All four test jobs use both of these non-existent references, so every test job will fail immediately at runtime with an "unable to resolve action" error, making the entire workflow non-functional. The rest of the repository correctly uses @v4 for these actions.

Additional Locations (1)

.github/workflows/run-system-tests.yml#L281-L282

^{Reviewed by Cursor Bugbot for commit 809f391. Configure here.}

sonarqubecloud · 2026-05-07T15:05:56Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

test: add nightly run flow for system tests

1db9598

pnarayanaswamy marked this pull request as ready for review April 29, 2026 12:23

pnarayanaswamy requested a review from a team as a code owner April 29, 2026 12:23

metamaskbotv2 Bot added the team-qa QA team label Apr 29, 2026

pnarayanaswamy added no-changelog no-changelog Indicates no external facing user changes, therefore no changelog documentation needed and removed team-qa QA team labels Apr 29, 2026

github-actions Bot added the size-L label Apr 29, 2026

cursor Bot reviewed Apr 29, 2026

View reviewed changes

Comment thread .github/workflows/run-system-tests.yml Outdated

pnarayanaswamy added 2 commits April 29, 2026 15:37

test: address review

5dda0dc

Merge branch 'main' into run-system-tests

9af5eee

cursor Bot reviewed Apr 29, 2026

View reviewed changes

Comment thread .github/workflows/run-system-tests.yml Outdated

fix: cursor bot

533eca2

cursor Bot reviewed Apr 29, 2026

View reviewed changes

Comment thread builds.yml

fix: add scripts

7c9bf6d

pnarayanaswamy added the team-qa QA team label Apr 29, 2026

jvbriones reviewed Apr 29, 2026

View reviewed changes

Comment thread .github/workflows/run-system-tests.yml

jvbriones requested changes Apr 29, 2026

View reviewed changes

fix: address review comments

3ed0ccc

cursor Bot reviewed Apr 30, 2026

View reviewed changes

Comment thread .github/workflows/run-system-tests.yml Outdated

test: fix permissions

58c74f1

pnarayanaswamy had a problem deploying to build-rc April 30, 2026 08:48 — with GitHub Actions Failure

pnarayanaswamy had a problem deploying to build-rc April 30, 2026 08:48 — with GitHub Actions Error

fix: source branch

d01e743

cursor Bot reviewed Apr 30, 2026

View reviewed changes

Comment thread .github/workflows/run-system-tests.yml

pnarayanaswamy had a problem deploying to build-rc April 30, 2026 08:58 — with GitHub Actions Failure

pnarayanaswamy had a problem deploying to build-rc April 30, 2026 08:59 — with GitHub Actions Failure

test: remove push test

f1d7841

jvbriones previously approved these changes Apr 30, 2026

View reviewed changes

pnarayanaswamy enabled auto-merge April 30, 2026 11:37

pnarayanaswamy added this to PR review queue May 4, 2026

github-project-automation Bot moved this to Needs dev review in PR review queue May 4, 2026

pnarayanaswamy moved this from Needs dev review to Has approvals, needs CODEOWNER in PR review queue May 4, 2026

tommasini reviewed May 4, 2026

View reviewed changes

pnarayanaswamy added 2 commits May 6, 2026 14:07

Merge branch 'main' into run-system-tests

d2c0525

fix: adress script review

9bfb8da

pnarayanaswamy dismissed jvbriones’s stale review via 9bfb8da May 6, 2026 15:27

Merge branch 'main' into run-system-tests

809f391

cursor Bot reviewed May 7, 2026

View reviewed changes

tommasini approved these changes May 7, 2026

View reviewed changes

github-project-automation Bot moved this from Has approvals, needs CODEOWNER to Review finalised - Ready to be merged in PR review queue May 7, 2026

pnarayanaswamy added this pull request to the merge queue May 7, 2026

Merged via the queue into main with commit 69ab778 May 7, 2026
60 checks passed

pnarayanaswamy deleted the run-system-tests branch May 7, 2026 15:24

github-actions Bot locked and limited conversation to collaborators May 7, 2026

metamaskbotv2 Bot added the release-7.77.0 Issue or pull request that will be included in release 7.77.0 label May 7, 2026

Uh oh!

Conversation

pnarayanaswamy commented Apr 29, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Performance checks (if applicable)

Pre-merge reviewer checklist

Uh oh!

github-actions Bot commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jvbriones Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 7, 2026

🔍 Smart E2E Test Selection

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot May 7, 2026

Choose a reason for hiding this comment

Non-existent action versions break all test jobs

Uh oh!

sonarqubecloud Bot commented May 7, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pnarayanaswamy commented Apr 29, 2026 •

edited by cursor Bot

Loading

jvbriones Apr 29, 2026 •

edited

Loading